Systems and techniques for rating items

ABSTRACT

Systems and techniques for rating items are provided. A method for providing item ratings may include obtaining predicted ratings for the item, and providing a set of ratings for the item including user-provided ratings and the predicted ratings.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority and benefit under 35 U.S.C. §119(e) ofU.S. Provisional Patent Application Ser. No. 62/156,584, titled“Presentation and Recommendation of Products in an Online Marketplace”and filed on May 4, 2015 under Attorney Docket No. CTL-001PR, thecontent of which is incorporated by reference herein in its entirety.

COPYRIGHT STATEMENT

This disclosure, including the drawings, contains material that issubject to copyright projection. The copyright owner has no objection tothe facsimile reproduction by anyone of the patent document or thepatent disclosure as it appears in the Patent and Trademark Officepatent files or records, but otherwise reserves all copyrightswhatsoever.

FIELD OF INVENTION

This disclosure relates generally to systems and techniques for ratingitems. Some embodiments relate specifically to rating items inelectronic commerce and, more particularly, to generating predictedratings for items in an online marketplace.

BACKGROUND

The widespread availability of computer devices capable of communicatingwith each other through communication networks (e.g., the Internet)facilitates online shopping and electronic commerce (“e-commerce”). Forexample, consumers can shop for items (e.g., products and/or services)in online marketplaces. Purchased products can be shipped directly tothe consumers. For many consumers, online shopping can be moreconvenient and more efficient than visiting brick-and-mortar stores inperson.

The availability of networked computer devices also facilitates thesharing of ratings and reviews of items by consumers. For example, manyonline marketplaces permit users to submit ratings and reviews of items,which other users can view when deciding whether to purchase the items.

SUMMARY OF THE INVENTION

Although user-provided ratings of items in an online marketplace canassist consumers in assessing other consumers' opinions of an itembefore purchasing the item, user-provided ratings can be misleading.Consumers who make purchasing decisions based on misleading ratings datacan make sub-optimal purchasing decisions, to their own detriment. Forexample, the number of user-provided ratings for some items (e.g., itemsoffered for sale only recently) can be very low, and the distribution ofthe ratings for such items may not be representative of the actualdistribution of opinions of consumers who are familiar with the item.Thus, there is a need for ratings systems and techniques that moreaccurately represent the actual distribution of opinions of consumerswho are familiar with an item, even when the number of user-providedratings for the item is low. The inventors have recognized andappreciated that the accuracy of the distribution of ratings for an itemwith a low number of user-provided ratings can be enhanced bysupplementing the user-provided ratings with one or more predictedratings. For many items, the actual distribution of item ratings can beaccurately predicted (e.g., based on the item's type, price,manufacturer, etc.). As the number of user-provided ratings for an itemincreases, the number of predicted ratings can be reduced, until onlyuser-provided ratings for the item are displayed.

According to another aspect of the present disclosure, acomputer-implemented method is provided, including: obtaining one ormore user-provided ratings for an item, wherein the one or moreuser-provided ratings are provided by users of a computer system;determining whether the number of user-provided ratings for the itemexceeds a threshold number of ratings for the item; and based on adetermination that the number of user-provided ratings for the item doesnot exceed the threshold number of ratings: obtaining a plurality ofpredicted ratings for the item, and providing a set of ratings for theitem to a device of a user, wherein the set of ratings for the itemincludes the one or more user-provided ratings and the plurality ofpredicted ratings.

In some embodiments, a distribution of the predicted ratings for theitem substantially matches a predicted distribution of ratings for theitem. In some embodiments, the method further includes determining thepredicted distribution of ratings for the item based, at least in part,on a categorization of the item, a manufacturer of the item, a seller ofthe item, and/or a price of the item. In some embodiments, the methodfurther includes determining the predicted distribution of ratings forthe item based, at least in part, on actual distributions of ratings forone or more other items. In some embodiments, the item and the one ormore other items are in the same category (or collection) of items, aremanufactured by the same manufacturer, are sold by the same seller,and/or have prices in the same range of prices.

In some embodiments, the one or more user-provided ratings are one ormore first user-provided ratings, and the method further includes:obtaining one or more second user-provider ratings for the item;determining whether a combined number of the first and seconduser-provided ratings for the item exceeds the threshold number ofratings for the item; and based on a determination that the combinednumber of user-provided ratings for the item exceeds the thresholdnumber of ratings: providing a set of ratings for the item to a deviceof a user, wherein the set of ratings for the item consists of the oneor more first user-provided ratings and the one or more seconduser-provided ratings.

According to another aspect of the present disclosure, a system isprovided, including one or more computers programmed to performoperations including: obtaining one or more user-provided ratings for anitem, wherein the one or more user-provided ratings are provided byusers of a computer system; determining whether the number ofuser-provided ratings for the item exceeds a threshold number of ratingsfor the item; and based on a determination that the number ofuser-provided ratings for the item does not exceed the threshold numberof ratings: obtaining a plurality of predicted ratings for the item, andproviding a set of ratings for the item to a device of a user, whereinthe set of ratings for the item includes the one or more user-providedratings and the plurality of predicted ratings.

In some embodiments, a distribution of the predicted ratings for theitem substantially matches a predicted distribution of ratings for theitem. In some embodiments, the operations further include determiningthe predicted distribution of ratings for the item based, at least inpart, on a categorization of the item, a manufacturer of the item, aseller of the item, and/or a price of the item. In some embodiments, theoperations further include determining the predicted distribution ofratings for the item based, at least in part, on actual distributions ofratings for one or more other items. In some embodiments, the item andthe one or more other items are in the same category (or collection) ofitems, are manufactured by the same manufacturer, are sold by the sameseller, and/or have prices in the same range of prices.

In some embodiments, the one or more user-provided ratings are one ormore first user-provided ratings, and the operations further include:obtaining one or more second user-provider ratings for the item;determining whether a combined number of the first and seconduser-provided ratings for the item exceeds the threshold number ofratings for the item; and based on a determination that the combinednumber of user-provided ratings for the item exceeds the thresholdnumber of ratings: providing a set of ratings for the item to a deviceof a user, wherein the set of ratings for the item consists of the oneor more first user-provided ratings and the one or more seconduser-provided ratings.

According to another aspect of the present disclosure, a computerstorage medium is provided, the computer storage medium havinginstructions stored thereon that, when executed by a data processingapparatus, cause the data processing apparatus to perform operationsincluding: obtaining one or more user-provided ratings for an item,wherein the one or more user-provided ratings are provided by users of acomputer system; determining whether the number of user-provided ratingsfor the item exceeds a threshold number of ratings for the item; andbased on a determination that the number of user-provided ratings forthe item does not exceed the threshold number of ratings: obtaining aplurality of predicted ratings for the item, and providing a set ofratings for the item to a device of a user, wherein the set of ratingsfor the item includes the one or more user-provided ratings and theplurality of predicted ratings.

In some embodiments, a distribution of the predicted ratings for theitem substantially matches a predicted distribution of ratings for theitem. In some embodiments, the operations further include determiningthe predicted distribution of ratings for the item based, at least inpart, on a categorization of the item, a manufacturer of the item, aseller of the item, and/or a price of the item. In some embodiments, theoperations further include determining the predicted distribution ofratings for the item based, at least in part, on actual distributions ofratings for one or more other items. In some embodiments, the item andthe one or more other items are in the same category (or collection) ofitems, are manufactured by the same manufacturer, are sold by the sameseller, and/or have prices in the same range of prices.

In some embodiments, the one or more user-provided ratings are one ormore first user-provided ratings, and the operations further include:obtaining one or more second user-provider ratings for the item;determining whether a combined number of the first and seconduser-provided ratings for the item exceeds the threshold number ofratings for the item; and based on a determination that the combinednumber of user-provided ratings for the item exceeds the thresholdnumber of ratings: providing a set of ratings for the item to a deviceof a user, wherein the set of ratings for the item consists of the oneor more first user-provided ratings and the one or more seconduser-provided ratings.

Other aspects and advantages of some embodiments of the invention willbecome apparent from the following drawings and detailed description,which illustrate the principles of the invention, by way of exampleonly.

The foregoing Summary, including the description of motivations for someembodiments and/or advantages of some embodiments, is intended to assistthe reader in understanding the present disclosure, and does not in anyway limit the scope of any of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages of the some embodiments may be understood by referring to thefollowing description taken in conjunction with the accompanyingdrawings. In the drawings, like reference characters generally refer tothe same parts throughout the different views. Also, the drawings arenot necessarily to scale, emphasis instead generally being placed uponillustrating principles of some embodiments of the invention.

FIG. 1 shows a computer system, according to some embodiments;

FIG. 2 shows a user interface for accessing an online marketplace,according to some embodiments;

FIGS. 3A, 3B, and 3C show first, second, and third user interfaces,respectively, for encouraging a user to place an item into a virtualshopping cart, according to some embodiments;

FIGS. 4A and 4B show first and second user interfaces, respectively, forencouraging a user to initiate a checkout process in an onlinemarketplace, according to some embodiments;

FIG. 5 shows a user interface for viewing an item feed, according tosome embodiments;

FIGS. 6A, 6B, and 6C show first, second, and third user interfaces,respectively, for encouraging a user to save an item to a virtual wishlist, according to some embodiments;

FIGS. 7A, 7B, 7C, and 7D show first, second, third, and fourth userinterfaces, respectively, for submitting ratings of an item, accordingto some embodiments; and

FIGS. 8A, 8B, and 8C show first, second, and third user interfaces,respectively, for registering for an online marketplace, according tosome embodiments.

DETAILED DESCRIPTION

FIG. 1 illustrates an example computer system 100 suitable forimplementing and providing access to an online marketplace. A serversystem 122 provides functionality for implementing an onlinemarketplace. The server system 122 comprises software components anddatabases that can be deployed at one or more data centers 121 in one ormore geographic locations, for example. The server system 122 softwarecomponents comprise a front-end server 112, a catalog generator 114, anda rating module 116. The server system 122 can also include one or moresoftware components for inventory management and load balancing. Thesoftware components can comprise subcomponents that can execute on thesame or on different individual data processing apparatus. The serversystem 122 databases comprise a user data database 132, transaction datadatabase 134, and product data database 136. The databases can reside inone or more physical storage systems. The software components anddatabases will be described further below.

The front-end server 112 provides catalog content to users of the onlinemarketplace, facilitates transactions by the users, and manages useraccess to the online marketplace. The catalog generator 114 generatescatalog content (or “a feed”) for users of the online marketplace. Therating module 116 generates rating data for items (e.g., products and/orservices) in the online marketplace. The front-end server 112, thecatalog generator 114, and the rating module 116 will be furtherdescribed below.

Users of the online marketplace (e.g., user 151) use client devices(e.g., client device 150) to access the online marketplace implementedby the server system 122 through one or more data communication networks113 such as the Internet, for example. A client device is a dataprocessing apparatus (e.g., a smart phone, a tablet computer, a smartwatch, a personal computer, a game console, or an in-car media system).Other examples of client devices are possible. The user can access theonline marketplace from a user interface 154 of a client application 152running on the client device 150. The client application 152 can be aweb browser or a special-purpose software application (e.g., a “mobileapp”), for example. The user interface 154 is a graphical user interfacethat allows a user to interact with one or more catalogs of the onlinestore, make purchases, create wish lists, and/or log on to the onlinestore, for example. The user interface 154 will be further describedbelow.

By way of illustration, when the user uses the client application 152 toaccess the online marketplace, the client application 152 retrieves,from the front-end server 112, catalog content specific for the user,and presents the catalog content in the user interface 154. The catalogcontent can be created by the catalog generator 114 and provided to thefront-end server 112. Creating catalog content will be further describedbelow.

When the user selects an item (e.g., product or service) to purchasefrom one or more catalogs presented in the user interface 154, theclient application 152 presents in the user interface 154 a “shoppingcart” or list that contains the selected products and/or services.Meanwhile, the client application 152 can send the front-end server 112a request for creating or updating a copy of the user's shopping cartwhich is stored in transaction data database 134.

When the user makes a purchase (e.g., initiates a checkout for an itemin the user's shopping cart presented by the client device 152), theclient device 152 sends the front-end server 112 a request forcompleting the purchase. The front-end server 112 completes the purchaseand stores a transaction record for the purchase in the transaction datadatabase 134.

The user can also select a product or service in one or more catalogspresented by the client application 152 to add the item to a “wish list”to which the user can return later for future viewing or purchasing. Inresponse to the user's selection, the client application 152 presents inthe user interface 152 a wish list that contains the selected productand/or service along with any previously added products and services.The client application 152 sends the front-end server 112 a request forcreating or updating a copy of the user's wish list which is stored inthe transaction data database 134.

Hybrid User Interface

FIG. 2 illustrates an example user interface 201 of the clientapplication 152 for accessing the online marketplace implemented by theserver system 122. The user interface 201 presents a product catalog (or“feed”) for the online marketplace. In some implementations, the userinterface 201 comprises a category panel 210, collection panel 220, andan item panel (e.g., a product panel) 230. The category panel 210presents one or more selectable icons corresponding to respective itemcategories in the online marketplace. Example categories includefashion, watchers, shores, shorts, shirt, jeans, sunglasses, jackets,polo, belt, pants, gadgets, hoodies, suits, rings, wallets and bags,services, and gadgets. Other categories are possible. The category panel210 can be scrolled horizontally (toward left or right) to displayavailable icons (categories). A user can select a category icon, causingthe client application 152 to present another user interface displayinga list of individual products (a product feed) of the correspondingcategory.

The collection panel 220 presents one or more collections of items(“item collections”, e.g., product collections or service collections).A product collection is a list of similar products. Each selectableframe (e.g., 221) displays sample product images of a correspondingcollection. The collection panel 220 can be scrolled horizontally todisplay available collections. A user can select a collection frame,causing the client application 152 to present another user interfacedisplaying a list of individual products (a product feed) of thecollection.

The product panel 230 presents a list or feed of individual products.Each selectable frame (e.g., 231) displays a sample image of acorresponding product. The product panel 230 can be scrolled vertically(up or down) to display available products. The product panel 230 canexpand upwards and overlay the collection panel 220 and category panel210. A user can select a product frame, causing the client application152 to present another user interface (a detail product page) displayingfurther information of the corresponding product.

The collections and product feeds described above are created for aparticular user (i.e., user-specific) by the catalog generator 114 ofthe server system 122. The client application 152 can record the user'sviewing history and send the viewing history to the front-end server112, which can in turn store the viewing history in the user datadatabase 132. The catalog generator 114 can access the viewing historyof the user and create catalog content including layout of the userinterface 201 for the user based on the user's viewing history, andprovide the catalog content to the client device 152 (e.g., via thefront-end server 112). For instance, if the user has viewed more oftenproducts or collections related to a particular category, the particularcategory can be placed ahead of other categories in the category panel210. Collections related to the particular category can also be placedahead of other collections in the collection panel 220. Products relatedto the particular category can also be placed ahead of products relatedto other categories in the product panel 230. In variousimplementations, the order of the collection or product placements inthe collection panel 220 and product panel 230 can be rearranged(shuffled) such that the user can discover “new” products. For instance,for the first 10 products listed in the product panel 230, the cataloggenerator 114 can create a layout that includes 4 products from acategory most viewed by the user, 2 products from a category viewed nextmost by the user, 2 products from a category viewed next most by theuser, and 2 products from one or more other categories (e.g., randomlychosen categories). The viewing history can be collected over arelatively short moving time window (e.g., past two weeks). In this way,content of a product feed can reflect more recent activities of the userin the online marketplace. Creation of product feeds will be furtherdescribed later.

Add to Cart Offers

To encourage a user to make a purchase through the online marketplaceimplemented by the server system 122, product feeds created by thecatalog generator 114 and presented by the client application 152 caninclude various incentives or offers for the user.

FIGS. 3A-3C illustrate user interfaces for encouraging a user to place aviewed product into a shopping cart. In some embodiments, the userinterfaces illustrated in FIGS. 3A-3C may be presented to the user aspart of method for encouraging the user to place a viewed product in ashopping cart. In FIG. 3A, a user can select a particular item (e.g.,product) 302 in an item feed (e.g., product feed) 301. In response tothe user's selection, the client application 152 presents an item detailpage (e.g., product detail page) 310, as illustrated in FIG. 3B. Theproduct detail page 310 includes an offer 312 for the user. The offer312 encourages the user to place the particular product in a shoppingcart to obtain (“unlock”) a discount to the listed price ($14) of theparticular product. The actual discount is unknown to the user beforethe user places the particular product in the shopping cart. The offer312 also includes a timer (“03:58”) indicating time left before theoffer expires, encouraging the user to place the particular product inthe shopping cart soon. After the user selects a “Buy” icon 314, theclient application 152 presents a shopping cart page 320, including anoverlaying frame 322, as illustrated in FIG. 3C. The overlaying frame322 indicates the discounted price ($13). The overlaying frame 322 alsoincludes a timer (“00:59:58”) indicating time left before the offerexpires, encouraging the user to purchase the particular product bycompleting a checkout of the particular product from the shopping cart.

In the product feed 301, a subset of products can be selected (e.g., bythe catalog generator 114) to provide offers encouraging the user toplace the products in the shopping cart. More particularly, the subsetof products can be selected randomly. The amount of discount for eachoffering can also be determined randomly. In this way, each user couldsee different products offering different discounts encouraging the eachuser to place products in the shopping cart. In addition, if theoffering of a particular product to a particular user expires, theoffering on the same particular product can be made unavailable to thatuser for a specified period of time (e.g., one week).

Checkout Offer

FIGS. 4A and 4B illustrate user interfaces for encouraging a user toinitiate a checkout of a product in a shopping cart. In someembodiments, the user interfaces illustrated in FIGS. 4A and 4B may bepresented to the user as part of method for encouraging the user toinitiate a checkout of a product in a shopping cart. FIG. 4A illustratesan example product detail page 401. In response to a user's selection ofa “Buy” icon 402 in the product detail page 401, the client application152 presents a shopping cart page 420, as illustrated in FIG. 4B. Theshopping cart page includes an offer 422 (“5% back as a gift card foryour next order”) that encourages the user to check out the product(from the shopping cart) within a specified time period, as indicated bya timer (“10:34”). The user can complete the purchase by selecting a“Checkout” icon 425 in the shopping cart page 420. The offer can beprovided to the user automatically. For instance, after receiving fromthe client application 152 a message indicating initiation of thepurchase by the user, the front-end server 152 can complete the purchaseand place the purchase information in the transaction data database 134.The front-end server 152 can also store in the transaction data database134 a record of a credit (5% of $12.00=$0.60) that can be used by theuser at a later time.

The offer for encouraging a user to check out a product in the user'sshopping cart can be provided to users that have not checked out (madepurchases) for a specified period of time (e.g., past two weeks). Theoffer can be provided to such users randomly, so a user cannot expectwhen the offer will be available. If the offer is shown to a particularuser and expires before the particular user makes a purchase, the offercan be made unavailable to the particular user for a specified period oftime (e.g., three days).

Hourly Deal

FIG. 5 illustrates an example item feed page (e.g., product feed page)501 with limited time offers. In FIG. 5, one or more products 502 fromthe product feed of the product feed page 501 have limited time offers(“Hourly Deals”). Each discounted product 502 has a timer (e.g.,“58:32”) indicating the time remaining before the offer expires. Thediscounted products 502 can be placed at the top of the product feed,and remain on top of the product feed after a refresh of the productfeed page 501 (before the timer expires).

The discounted products 502 can be selected (e.g., by the cataloggenerator 114) randomly from the product feed. When a particular productis selected for this offer, the particular product can be madeunavailable for selection for this offer for a specified period of time(e.g., five days).

Save to See Price

FIGS. 6A-6C illustrate user interfaces for encouraging a user to save aproduct to a wish list. In some embodiments, the user interfacesillustrated in FIGS. 6A-6C may be presented to the user as part ofmethod for encouraging the user to save a product to a wish list. FIG.6A illustrates an item feed page (e.g., a product feed page) 601. In theproduct feed page 601, a discount (−38%) is shown for a product 602 butthe price for the product 602 is not shown. Instead, a message “Save tosee today's price!” encourages the user the save the product 602 to awish list. Note that even after the user selects the product 602 in theproduct feed page 601, without selecting the “Save” icon 605 (to savethe product 602 to a wish list), the detail item page (e.g., detailproduct page) 611 still does not show the price for the product 602, asillustrated in FIG. 6B. Instead, the detail product page 611 shows amessage (“Save to see today's price!”) that encourages the user the savethe product 602 to a wish list. In the example of FIG. 6B, the detailproduct page 611 also include a timer 615 (“04:32”) that indicates timeremaining for the offer, and encourages the user to add the product 612to the shopping cart.

If the user selects the save icon 605 in the product feed page 601illustrated in FIG. 6A, or selects the save icon 614 in the detailproduct page 611 illustrated in FIG. 6B, the client application 152updates the detail product page 611 with the discounted price 621 ($31),as illustrated in FIG. 6C. In the example of FIG. 6C, the updated detailproduct page 611 also includes the timer 615 encouraging the user toplace the product 612 in the shopping cart within the remaining time.

In various implementations, one product is shown at a time with asave-to-see-price offer in a view of a product feed page. In referenceto FIG. 6A, since approximately four to six products are shown at a timein a view port of the product feed page 601, one out of every four tosix products in the product feed page 601 can be shown with asave-to-see price offer, for example. Within a group of four products, aproduct with the largest discount (e.g., largest difference between amanufacture suggested retail price and an actual retail price) can beselected to hide its retail price and be shown with a save-to-see-priceoffer.

User Submitted Rating Photos and Inferred Ratings

FIGS. 7A-7D illustrate user interfaces through which users can submit orview ratings of items (e.g., products) the users purchased through theonline marketplace implemented by the server system 122. In someembodiments, the user interfaces illustrated in FIGS. 7A-7D may bepresented to the user as part of method for submitting or viewingratings of items. FIG. 7A illustrates an example item order history page(e.g., product order history page) 701 presented by the clientapplication 152. In response to user selection of an “Edit Rating” icon705, the client application 152 presents a rating page 711, asillustrated in FIG. 7B. In the rating page 711, the user has the option(715) to upload a photo of the actual product the user received. Forinstance, the use can upload a photo of the product (e.g., from theclient device 150) through the rating page 711. After receiving anindication from the user about the photo, the client application 152transmits the photo and information for the user and the purchase (e.g.,a particular size of the product the user purchased, an identifier forthe purchase record stored in the transaction data database 134, etc.)to the front-end server 112. The front-end server 112 can place thephoto provided by the user in the product data database 136.Additionally, the photo can be reviewed (e.g., for picture quality) by acomputer or a person. The catalog generator 114 can generate a productdetail page for this particular product, and include the user providedphotos with other images of the particular product in the product detailpage.

In FIG. 7C, when another user views a detail product page 750 for thisparticular product, the user can slide left or right on an image 751 ofthe product to see other images, including user-supplied photos, of theparticular product. The user can also select a “see all” icon 752 to seeother images of the particular product. FIG. 7D illustrates the detailproduct page 750 including a user-supplied photo for the particularproduct. The detail product page 750 can also include information 761related to the user-supplied photo, for example, the user who suppliedthe photo, the size of the product for this photo, etc.

Users can also provide ratings in scores such as 1-star, 2-star, 3-star,4-star, and 5-star. A 5-star rating can be the best rating, while a1-star rating can be the worst rating. For example, the user can submita rating (score) of a particular item (e.g., product) through a userinterface (e.g., 701) of the client application 152. The clientapplication 152 transmits the user's rating to the front-end server 112,which in turns stores the rating (in reference to the particularproduct) in the product data database 136. When the catalog generator114 creates a detail product page for the particular product, the ratingmodule 116 access the product data database 136 for rating data for theparticular product and provides the overall rating data (e.g., one5-star, four 3-star, one two-star, one 1-star) to the catalog generator114 to be included in the detail product page.

When the number of ratings (scores) submitted by users on a particularproduct is low (e.g., less than 10), the overall rating data on theparticular data can be skewed toward one end of the rating scale (e.g.,mostly 1-star scores). To prevent the skewing, the rating module canapply a pre-defined distribution to the overall rating data when thenumber of ratings submitted is low. The overall rating data isautomatically adjusted when more rating scores are submitted andeventually reflects actual overall rating data submitted by users. Forinstance, the pre-defined distribution can be 20% 5-star, 20% 4-star,10% 3-star, 20% 2-star, 30% 1-star. The rating module can calculateinitial overall rating data assuming a certain number of users (e.g.,100) who may have purchased the particular product but not submittedratings on the particular product. Thus the overall rating data can be20 5-star, 20 4-star, 10 3-star, 20 2-star, and 30 1-star. When thereare additional scores submitted by users, the rating module adds thesubmitted scores (counts) to the initial overall rating data for theoverall rating data.

In some embodiments, the system 100 may perform a method M1 for reducingor preventing skewing of ratings for an item. In a first step of themethod M1, the server system 122 may obtain one or more user-providedratings for an item. Users may enter the ratings into a user interface154 of a client application 152 executing on a client device 150. Theuser interface 154 may include one or more of the user interfacesillustrated in FIGS. 7A-7D, or any other user interface suitable forproviding ratings of items (e.g., star-based ratings on a scale of1-star to 5-stars). The client device 150 may send the user-providedratings to the server system 122.

In a second step of the method M1, the server system 122 may determinewhether the number of user-provided ratings for the item exceeds athreshold number of ratings for the item. The threshold number ofratings may be 5, 10, 20, 50, or any other suitable number of ratings.

In a third step of the method M1, based on a determination that thenumber of user-provided ratings for the item does not exceed thethreshold number of ratings, the server system 122 obtains predictedratings for the item (e.g., the “initial overall rating data” describedabove) and provides a set of ratings for the item (e.g., the “overallrating data” described above) to a client device 150 of a user. A clientapplication 152 of the client device 150 may display the set of ratingsfor the item in a user interface 154. The set of ratings provided to theuser may include any user-provided ratings for the item and thepredicted ratings for the item.

In some embodiments, the probability distribution of the predictedratings for the item substantially matches a predicted distribution ofratings for the item. As used herein, two probability distributions“substantially match” if the mean, median, variance, and standarddeviation of the first distribution, respectively, are within plus orminus 20% of the mean, median, variance, and standard deviation of thesecond distribution. In some embodiments, the predicted distribution ofratings for the item may be determined based on the item's category,manufacturer, seller, and/or price. For example, the predicteddistribution of ratings for the item may be obtained by combining (e.g.,averaging) the actual distributions of ratings for other items in thesame category having approximately the same price as the item ofinterest.

In a fourth step of the method M1, the server system 122 may obtain oneor more additional user-provided ratings for the item, and may determinethat the total number of user-provided ratings for the item exceeds thethreshold number of ratings. Based on this determination, the serversystem 122 may provide a set of ratings for the item to a client device150 of a user. The provided set of ratings may include the user-providedratings for the item, but not the predicted ratings for the item. Inother words, when the number of user-provided ratings for an itemexceeds the threshold, the server system 122 may discontinue using thepredicted ratings for the item.

Signup Flow

FIG. 8A-8C illustrate user interface for registering (e.g., “signingup”) for the online marketplace implemented by the server system 122,using the client application 152. In some embodiments, the userinterfaces illustrated in FIGS. 8A-8C may be presented to the user aspart of method for registering for the online marketplace. FIG. 8Aillustrates an example introduction page (“splash page”) presented bythe client application 152. A user can select “Create Account” toregister for the online marketplace. In response to the user'sselection, the client application 152 presents a signup form pageillustrated in FIG. 8B. The user can provide a name, email address, andpassword to create an account with the online marketplace. In addition,the user can provide an age, gender, and location to the onlinemarketplace. The client application 152 presents a collections pageillustrated in FIG. 8C. The user can select a particular collection fromthe collection page. The client application 152 transmits to thefront-end server 112 the user information provided by the user. Thefront-end server 112 stores the user information in the user datadatabase 132.

In some implementations, the particular category selected by the userduring the signup is initially the first (and only) record in the user'sview history. In response to a request from the client application 152for catalog content and layout—e.g., when the user enters the userinterface 201 described in reference to FIG. 2 after selecting theparticular category in the collections page of FIG. 8C, the cataloggenerator 114 can create a layout for the user interface 201 such thatthe particular collection is placed in front of other collections in thecollection panel 220. A category related to the particular collectioncan also be placed in front of other categories in the category panel210.

User Similarity and Product Feeds

The catalog generator 114 can create an item feed (e.g., a product feed)(e.g., for a category or collection) based on popularity of productsand/or services. For instance, a product with higher sales volume (morepopular) can be ranked higher in a product feed than another productwith lower sale volume (less popular). Higher ranked products can appearbefore lower ranked products in a given feed. Products can be ranked by“purchaseability score” (described below), which may be the likelihoodthat a particular user would purchase the given product. As anotherexample, a product with a higher number of instances of being placed inusers' wish lists (more popular) can be ranked higher in a product feedthan another product with a lower number of instances being placed inusers' wish lists (less popular).

In various implementations, the purchaseability score of a product canbe, for example, purchase volume for the product divided by a weightedsum of respective sums of different user actions. User actions can bethat a user views the product (e.g., the product's detail product page),a user places the product in wish lists, a user places the product in ashopping cart, and/or a user purchases the product. Other types of useractions are possible.

As another example, the purchaseability score of a product can be aweighted sum of the number of instances of particular user actions forthe product, normalized to a number of impressions of the product. Thenumber of impressions can be a number of the products being viewed byusers in product feed pages during a specified time period (e.g., pastthree days). In other implementations, the number of impressions can bea number of the product's detailed product page being viewed by usersduring a specified time period. In some embodiments, “impressions” of anitem include any suitable events or acts whereby information about theitem (e.g., an advertisement for the item, an image of the item, adetailed item page for the item, etc.) is presented (e.g., displayed)via a user interface of a client device 150. For purposes of determiningthe purchaseability score for an item, the number of impressions of theitem may be limited to impressions generated during a specified timeperiod.

For instance, the purchaseability score of a product can be as follows:

${{10000 \times \frac{purchases}{impressions} \times \sqrt{price}} + {0.1 \times \frac{{added}\mspace{14mu} {to}\mspace{14mu} {shopping}\mspace{14mu} {carts}}{impressions}} + {0.001 \times \frac{{saved}\mspace{14mu} {to}\mspace{14mu} {wish}\mspace{14mu} {list}}{impressions}}},$

where “purchases” can refer to the number of users who purchased theproduct or to the total quantity of the product purchased by a specifiedgroup of users (e.g., all users), “added to shopping carts” can refer tothe number of users who added the product to their shopping carts or tothe total quantity of the product added to shopping carts of a specifiedgroup of users, and “saved to wish list” can refer to the number ofusers who added the product to their wish lists or to the total quantityof the product added to wish lists of a specified group of users.

Additionally or in the alternative, a product feed for a particular usercan be created based on the particular user's similarity to anothergroup of users. In addition to user segments (e.g., age, gender,location), similarity between users can also be determined by useractions. For instance, similar users can have similar purchase volumeduring a specified time period (e.g., past week). Similar users can havesimilar items placed in their respective wish lists. The cataloggenerator 114 can create a product feed for a particular user by copyingor aggregating one or more product feeds of a group of users that havesimilar age, gender, location, or user actions as the particular user.As another example, weights in the purchaseability score described abovecan be adjusted based on corresponding weights in the purchaseabilityscore formulas of similar users.

In particular implementations, assume that A is a sparse matrix suchthat A_(i,j)=1 when user i is engaged with product j, and A_(i,j)=0otherwise. A user i may be engaged with a product j if an impression ofthe product has been presented (e.g., displayed) to the user via theclient device 150 (e.g., within a specified time period), if the userhas added the product to the user's shopping cart, if the user has addedthe product to the user's wish list, and/or if the user has purchasedthe product. The similarity score between users u and v can be:

$S_{u,v} = \frac{\sum\limits_{j}\; {A_{u,j} \times A_{v,j}}}{\sqrt{\sum\limits_{j}\; A_{u,j}^{2}} \times \sqrt{\sum\limits_{j}\; A_{v,j}^{2}}}$

For a given user u, the catalog generator 114 can find top similar users(e.g., top 10,000 similar users) by respective similarity scores. Thecatalog generator 114 can calculate a similarity weighted interestedscore for a given product with respect to the top similar users:

$\sum\limits_{k}\; {SW}_{u,k}$

In the above equation, SW_(u,k)=S_(u,k) if a top similar user kpurchased the given product, and SW_(u,k)=0 otherwise. The cataloggenerator 114 can determine a set (e.g., 1,000) of top ranked productsaccording to the above equation, and recommend the set of top rankedproducts to user u.

The catalog generator 114 can create, for a particular user, a first setof items (e.g., products) based on purchaseability and a second set ofitems (e.g., products) based on similarity scores as described above.The catalog generator 114 can generate a product feed for the particularuser by interleaving the first and second sets of products. Forinstance, when the product feed is displayed to the particular user forthe first time, products from the first set can have 70% opportunity ofbeing displayed in a given placement of the product feed, until thefirst set is exhausted and all products from the second set are thendisplayed. When the product feed is displayed to the particular user forthe second time (e.g., due to a refresh by the particular user),products from the first set can have 50% opportunity of being displayedin a given placement of the product feed. The ratio can be reduced to30% for the third time the product feed is displayed to the particularuser and 10% for the fourth time the product feed is displayed to theparticular user. Subsequently, products from the first set can be placedrandomly between products from the second set in the product feed.

When the user requests or refreshes an item feed (e.g., a product feed),the catalog generator 114 or the client application 152 can rearrange(shuffle) the order of products in the product feed. In this way, theuser can see something new every time the user loads or refresh theproduct feed. In various implementations, the order of the products inthe product feed can be rearranged by a weighted random shuffling orordering, for example:

$\frac{\log ( {{random}( {0,1} )} )}{{purchaseability}\mspace{14mu} {score}}.$

In some embodiments, the system 100 may perform a method M2 fordetermining the order of items in an item feed. In a first step of themethod M2, the server system 122 may receive data indicative ofselection of a type of item (e.g., a category of items and/or acollection of items). The selection data may be received, for example,from a client device 150 of a user. For example, a user may select acategory via the category panel 210 of the user interface shown in FIG.2, or via any other suitable user interface. As another example, a usermay select a collection via the collection panel 220 of the userinterface shown in FIG. 2, or via any other suitable user interface. Theclient device 150 may then send data indicative of the selected itemtype to the server system 122.

In a second step of the method M2, the server system 122 may determinepurchaseability scores for a set of items of the specified type (e.g.,for a set of items in the specified category or collection). The serversystem 122 may determine an item's purchaseability score based, at leastin part, on (1) the number of impressions of the item provided to aspecified group of users (e.g., all users) during a specified timeperiod, (2) the quantity of the item purchased by the specified group ofusers, and (3) the quantity of the item added to shopping carts of thespecified group of users. In some embodiments, the server system'sdetermination of the purchaseability score for an item is also based onthe purchase price of the item and/or on the total quantity of the itemadded to the wish lists of the specified group of users. In someembodiments, the server system 122 determines the purchaseability scorefor an item based, at least in part, on the expression:

${W_{1} \times \frac{quant\_ purchases}{impressions} \times \sqrt{price}} + {W_{2} \times \frac{quant\_ cart}{impressions}} + {W_{3} \times \frac{{quant\_ wish}{\_ list}}{impressions}}$

wherein W₁, W₂, and W₃ are first, second, and third weights,respectively, wherein ‘impressions’ is the number of impressions of therespective item provided to a specified group of users during aspecified time period (or the number of users to whom impressions of theitem were presented during the time period), wherein ‘quant_purchases’is the quantity of the item purchased by the specified group of users(or the number of users in the group who purchased the item), wherein‘quant_cart’ is the quantity of the item added to shopping carts of thespecified group of users (or the number of users in the group who addedat least one of the item to their shopping carts), wherein‘quant_wish_list’ is the quantity of the item added to wish lists of thespecified group of users (or the number of users who added at least oneof the item to their wish lists), and wherein ‘price’ is a purchaseprice of the item (e.g., an average purchase price of the item). Thespecified group of users may be selected by the server system 122 based,at least in part, on similarities between the group of users and theuser for whom the purchaseability scores are being calculated.

In a third step of the method M2, the server system 122 may determine aranking of the set of items of the specified type based, at least inpart, on the purchaseability scores of the respective items included inthe plurality of items. The ranking can be, for example, a total rankingor a partial ranking.

In a fourth step of the method M2, the server system 122 may determine aranking of a second set of items. The second set of items may includeitems contained in the item feeds of a group of users who are similar tothe user for whom the items are being ranked. In some embodiments, theserver system ranks the second set of items for the user based, at leastin part, on the similarities between the user and the group of userswhose item feeds provided the items. In some embodiments, thesimilarities between the user and the group of users may be determinedusing the formula provided above. In some embodiments, the similaritiesbetween users may include similarities in demographics (e.g., theabove-described “user segments”). In some embodiments, the similaritiesbetween users may include similarities in shopping activity (e.g.,viewing certain items, adding certain items to a shopping cart and/orwish list, purchasing certain items, etc.).

In some embodiments, the second set of items may consist entirely ofitems not contained in the first set of items. In some embodiments, thesecond set of items and the first set of items may partially overlap(e.g., one or more items may be contained in both sets of items). Inembodiments where the sets of items overlap, any item that is containedin both sets may be removed from either of the sets, to avoidduplication of items in the item feed.

In a fifth step of the method M2, the server system 122 may use thefirst set of items (and, optionally, the second set of items) topopulate the item feed for the type of item selected by the user. If theitem feed contains only items from the first set of items, the orderingof items in the item feed may match the ranking of the items in thefirst set of items. If the item feed contains items from the first andsecond sets of items, the ordering of items in the item feed may dependon the rankings of the first and second sets of items.

The server system 122 may merge the first and second sets of items intoan ordered item feed using any suitable technique. In some embodiments,first and second sets of items may be merged into the ordered item feedwithout disturbing the relative ranking among the items included in thefirst set of items, and without disturbing the relative ranking amongthe items included in the second set of items. Thus, the partialordering within the item feed of items contained in the first set ofitems may match the ranking of the same items within the first set, andthe partial ordering within the item feed of items contained in thesecond set of items may match the ranking of the same items within thesecond set. In some embodiments, the server system 122 mayprobabilistically determine the relative ordering within the item feedbetween items from the first set and items from the second set.

In some embodiments, the server system 122 determines the ordering ofthe items in the item feed by: (a) probabilistically selecting betweenthe first set of items and the second set of items, wherein a firstprobability of selecting the first set of items is P₁ and a secondprobability of selecting the second set of items is 1-P₁; (b) from theselected set of items, identifying the highest-ranked item that has notyet been assigned a position in the ordering of item feed, (c) assigningthe identified item to the next position in the ordering of the itemfeed; and (d) repeating steps (a)-(c) until a specified number of itemshave been assigned positions in the ordering of the item feed or untilall the items in the first and second sets of items have been assignedpositions in the ordering of the item feed. As described above, theserver system may change the first and second probabilities in responseto passage of a specified period of time and/or in response to a requestto change the ordering of the set of items. As further described above,the server system 122 may change the ranking of the items in the firstset of items by performing a weighted random shuffling of thepurchaseability scores of those items.

In a sixth step of the method M2, the server system 122 may provide theitem feed data to a client device 150 of the user. The client device 150may be configured to display a scrollable feed of item panels (e.g.,frames) corresponding to the items identified in the item feed data. Theitem panels in the scrollable feed may be ordered according to theordering of the corresponding items in the item feed data.

New Products

For a new product added to the online marketplace implemented by theserver system 122, the catalog generator 114 can insert the new productin an existing product feed for users to “explore.” The cataloggenerator 114 first assigns user segments for the new products based oncontextual information from the new product's description. For instance,the catalog generator 114 can assign the new product to a female usersegment if the product's description includes the phrase “hand bag” oranother phrase generally associated with items that are generally ofgreater interest to women than to men. The catalog generator 114 canassign the new product to a male user segment if the product'sdescription includes the phrase “oxford dress shoes” or another phrasegenerally associated with items that are generally of greater interestto men than to women. The catalog generator 114 can assign the newproduct to age segments younger than 41 years old if the product'sdescription includes “video game” or another phrase generally associatedwith items that are generally of greater interest to persons youngerthan 41 years old than to persons 41 years old or older. The cataloggenerator 114 then inserts the new product into one or more productfeeds related to the assigned user segments. As users views the newproducts and perform various user actions (e.g., viewing, placing in awish list, placing in a shopping cart, purchasing), the cataloggenerator 114 can place the new product in different product feeds basedon user actions as described earlier.

For a given item (e.g., product), the catalog generator 114 can maintainan engagement score (e.g., a ratio of a number of user actions to anumber of impressions for the product). When a number impressions of anew product exceeds a specified threshold (e.g., 10,000), the cataloggenerator 114 compares the new product's engagement score to engagementscores of other existing products in a user segment (e.g., a male usersegment). If the new product's engagement score is comparable to orhigher than engagement scores of the other existing products in the usersegment (e.g., the new product's engagement score is equal to or higherthan a median engagement score of the other products), the new productcan be kept for exploration by users. Otherwise the new product can beconsidered not interesting to users and the catalog generator 114 canstop showing the new product in product feeds. A new product (e.g., ahigh-quality new product) can eventually be included in product feedrecommendation based on similarity or purchase-ability scores asdescribed earlier.

Further Description of Some Embodiments

Although shopping in an online marketplace can be more convenient thatvisiting a brick-and-mortar store, many online marketplaces provide aninefficient online shopping experience, thereby making inefficient useof consumers' time and of the resources of computer systems thatimplement and provide access to the online marketplaces. Suchinefficiency can be particularly acute when the consumer is shopping foran item of a particular type, but has not yet decided which item topurchase. The inventors have recognized and appreciated that theefficiency of a user's online shopping experience can be enhanced bypresenting items of a specified type to a user in a particular order,based on the items' “purchaseability scores.” In this way, the itemsthat the user is most likely to purchase (or to be interested inpurchasing) may be presented to the user before (or in lieu of)presenting other items that the user is less likely to purchase. As aresult, the user may successfully complete the shopping experience morequickly, thereby saving the user's time and conserving the resources ofthe computer systems that implement and provide access to the onlinemarketplace.

According to an aspect of the present disclosure, a computer-implementedmethod is provided, including: receiving, from a device of a user, dataindicative of selection of a type of item; for each of a plurality ofitems of the type, determining a purchaseability score for therespective item based, at least in part, on (1) a number of impressionsof the respective item provided to a plurality of users during a timeperiod, (2) a quantity of the respective item purchased by the pluralityof users, and (3) a quantity of the respective item added to shoppingcarts of the plurality of users; determining a ranking of the itemsincluded in the plurality of items based, at least in part, on thepurchaseability scores of the respective items included in the pluralityof items; generating item feed data indicating an ordering of a set ofitems including the plurality of items, wherein the ordering of theitems included in the set of items is determined based, at least inpart, on the ranking of the items included in the plurality of items;and providing the item feed data to the device of the user, wherein thedevice of the user is configured to display a scrollable feed of itempanels corresponding to the items included in the set of items, andwherein the item panels in the scrollable feed are ordered according tothe ordering of the corresponding items in the item feed data.

In some embodiments, the purchaseability score for the respective itemis further based, at least in part, on a purchase price of the item. Insome embodiments, the purchaseability score for the respective item isfurther based, at least in part, on a quantity of the respective itemadded to wish lists of the plurality of users.

In some embodiments, the purchaseability score for the respective itemis based, at least in part, on the expression:

${W_{1} \times \frac{quant\_ purchases}{impressions} \times \sqrt{price}} + {W_{2} \times \frac{quant\_ cart}{impressions}} + {W_{3} \times \frac{{quant\_ wish}{\_ list}}{impressions}}$

wherein W₁, W₂, and W₃ are first, second, and third weights,respectively, wherein ‘impressions’ is the number of impressions of therespective item provided to the plurality of users during the timeperiod, wherein ‘quant_purchases’ is the quantity of the item purchasedby the plurality of users, wherein ‘quant_cart’ is the quantity of theitem added to shopping carts of the plurality of users, wherein‘quant_wish_list’ is the quantity of the item added to wish lists of theplurality of users, and wherein ‘price’ is a purchase price of the item.

In some embodiments, the method further includes selecting users forinclusion in the plurality of users based, at least in part, on one ormore similarities between the user and the users. In some embodiments,the plurality of items is a first plurality of items, the method furtherincludes determining a ranking of a second plurality of items based, atleast in part, on (1) the one or more similarities between the user andthe users, and (2) data indicating whether the users purchased therespective items included in the plurality of items, and the set ofitems further includes the second plurality of items. In someembodiments, the ordering of the set of items is further determinedbased, at least in part, on the ranking of the second plurality ofitems.

In some embodiments, an ordering among the first plurality of itemswithin the set of items matches an ordering among the first plurality ofitems within the ranking of the first plurality of items, an orderingamong the second plurality of items within the set of items matches anordering among the second plurality of items within the ranking of thesecond plurality of items, and an ordering within the set of itemsbetween items included in the first plurality of items and itemsincluded in the second plurality of items is determinedprobabilistically.

In some embodiments, the method further includes determining theordering of the set of items, including: (a) probabilistically selectingbetween the first plurality of items and the second plurality of items,wherein a first probability of selecting the first plurality of items isP₁ and a second probability of selecting the second plurality of itemsis 1-P₁; (b) from the selected plurality of items, identifying ahighest-ranked item that has not yet been assigned a position in theordering of the set of items, and assigning the identified item to anext position in the ordering of the set of items; and repeating steps(a) and (b) until a specified number of items have been assignedpositions in the ordering of the set of items or until all the items inthe first and second pluralities of items have been assigned positionsin the ordering of the set of items.

In some embodiments, the method further includes changing the first andsecond probabilities in response to passage of a specified period oftime and/or in response to a request to change the ordering of the setof items. In some embodiments, the method further includes changing therankings of the items included in the plurality of items by performing aweighted random shuffling of the purchaseability scores of the itemsincluded in the plurality of items.

According to another aspect of the present disclosure, a system isprovided, including one or more computers programmed to performoperations including: receiving, from a device of a user, dataindicative of selection of a type of item; for each of a plurality ofitems of the type, determining a purchaseability score for therespective item based, at least in part, on (1) a number of impressionsof the respective item provided to a plurality of users during a timeperiod, (2) a quantity of the respective item purchased by the pluralityof users, and (3) a quantity of the respective item added to shoppingcarts of the plurality of users; determining a ranking of the itemsincluded in the plurality of items based, at least in part, on thepurchaseability scores of the respective items included in the pluralityof items; generating item feed data indicating an ordering of a set ofitems including the plurality of items, wherein the ordering of theitems included in the set of items is determined based, at least inpart, on the ranking of the items included in the plurality of items;and providing the item feed data to the device of the user, wherein thedevice of the user is configured to display a scrollable feed of itempanels corresponding to the items included in the set of items, andwherein the item panels in the scrollable feed are ordered according tothe ordering of the corresponding items in the item feed data.

In some embodiments, the purchaseability score for the respective itemis further based, at least in part, on a purchase price of the item. Insome embodiments, the purchaseability score for the respective item isfurther based, at least in part, on a quantity of the respective itemadded to wish lists of the plurality of users.

In some embodiments, the purchaseability score for the respective itemis based, at least in part, on the expression:

${W_{1} \times \frac{quant\_ purchases}{impressions} \times \sqrt{price}} + {W_{2} \times \frac{quant\_ cart}{impressions}} + {W_{3} \times \frac{{quant\_ wish}{\_ list}}{impressions}}$

wherein W₁, W₂, and W₃ are first, second, and third weights,respectively, wherein ‘impressions’ is the number of impressions of therespective item provided to the plurality of users during the timeperiod, wherein ‘quant_purchases’ is the quantity of the item purchasedby the plurality of users, wherein ‘quant_cart’ is the quantity of theitem added to shopping carts of the plurality of users, wherein‘quant_wish_list’ is the quantity of the item added to wish lists of theplurality of users, and wherein ‘price’ is a purchase price of the item.

In some embodiments, the operations further include selecting users forinclusion in the plurality of users based, at least in part, on one ormore similarities between the user and the users. In some embodiments,the plurality of items is a first plurality of items, wherein theoperations further include determining a ranking of a second pluralityof items based, at least in part, on (1) the one or more similaritiesbetween the user and the users, and (2) data indicating whether theusers purchased the respective items included in the plurality of items,and wherein the set of items further includes the second plurality ofitems. In some embodiments, the ordering of the set of items is furtherdetermined based, at least in part, on the ranking of the secondplurality of items.

In some embodiments, an ordering among the first plurality of itemswithin the set of items matches an ordering among the first plurality ofitems within the ranking of the first plurality of items, an orderingamong the second plurality of items within the set of items matches anordering among the second plurality of items within the ranking of thesecond plurality of items, and an ordering within the set of itemsbetween items included in the first plurality of items and itemsincluded in the second plurality of items is determinedprobabilistically.

In some embodiments, the operations further include determining theordering of the set of items, including: (a) probabilistically selectingbetween the first plurality of items and the second plurality of items,wherein a first probability of selecting the first plurality of items isP₁ and a second probability of selecting the second plurality of itemsis 1-P₁; (b) from the selected plurality of items, identifying ahighest-ranked item that has not yet been assigned a position in theordering of the set of items, and assigning the identified item to anext position in the ordering of the set of items; and repeating steps(a) and (b) until a specified number of items have been assignedpositions in the ordering of the set of items or until all the items inthe first and second pluralities of items have been assigned positionsin the ordering of the set of items.

In some embodiments, the operations further include changing the firstand second probabilities in response to passage of a specified period oftime and/or in response to a request to change the ordering of the setof items. In some embodiments, the operations further include changingthe rankings of the items included in the plurality of items byperforming a weighted random shuffling of the purchaseability scores ofthe items included in the plurality of items.

According to another aspect of the present disclosure, a computerstorage medium is provided, the computer storage medium havinginstructions stored thereon that, when executed by a data processingapparatus, cause the data processing apparatus to perform operationsincluding: receiving, from a device of a user, data indicative ofselection of a type of item; for each of a plurality of items of thetype, determining a purchaseability score for the respective item based,at least in part, on (1) a number of impressions of the respective itemprovided to a plurality of users during a time period, (2) a quantity ofthe respective item purchased by the plurality of users, and (3) aquantity of the respective item added to shopping carts of the pluralityof users; determining a ranking of the items included in the pluralityof items based, at least in part, on the purchaseability scores of therespective items included in the plurality of items; generating itemfeed data indicating an ordering of a set of items including theplurality of items, wherein the ordering of the items included in theset of items is determined based, at least in part, on the ranking ofthe items included in the plurality of items; and providing the itemfeed data to the device of the user, wherein the device of the user isconfigured to display a scrollable feed of item panels corresponding tothe items included in the set of items, and wherein the item panels inthe scrollable feed are ordered according to the ordering of thecorresponding items in the item feed data.

Representative Implementations

Implementations of the subject matter and the operations described inthis specification (including, but not limited to, the methods M1 andM2, and the operations performed by the client device 150, server system122, front-end server 112, catalog generator 114, and/or rating module116) can be implemented in digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. Implementations of the subject matter describedin this specification can be implemented as one or more computerprograms, i.e., one or more modules of computer program instructions,encoded on computer storage medium for execution by, or to control theoperation of, data processing apparatus. Alternatively or in addition,the program instructions can be encoded on an artificially-generatedpropagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storagemedium is not a propagated signal, a computer storage medium can be asource or destination of computer program instructions encoded in anartificially-generated propagated signal. The computer storage mediumcan also be, or be included in, one or more separate physical componentsor media (e.g., multiple CDs, disks, or other storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languageresource), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending resources to and receiving resources from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), an inter-network (e.g., theInternet), and peer-to-peer networks (e.g., ad hoc peer-to-peernetworks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someimplementations, a server transmits data (e.g., an HTML page) to aclient device (e.g., for purposes of displaying data to and receivinguser input from a user interacting with the client device). Datagenerated at the client device (e.g., a result of the user interaction)can be received from the client device at the server.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features that are described in this specification inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Having thus described several aspects of at least one embodiment of thisinvention, it is to be appreciated that various alterations,modifications, and improvements will readily occur to those skilled inthe art. Such alterations, modifications, and improvements are intendedto be part of this disclosure, and are intended to be within the spiritand scope of the invention. Accordingly, the foregoing description anddrawings are by way of example only.

Various aspects of the present disclosure can be used alone, incombination, or in a variety of arrangements not specifically describedin the foregoing, and the invention is therefore not limited in itsapplication to the details and arrangement of components set forth inthe foregoing description or illustrated in the drawings. For example,aspects described in one embodiment can be combined in a suitable mannerwith aspects described in other embodiments.

TERMINOLOGY

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting.

The term “approximately”, the phrase “approximately equal to”, and othersimilar phrases, as used in the specification and the claims (e.g., “Xhas a value of approximately Y” or “X is approximately equal to Y”),should be understood to mean that one value (X) is within apredetermined range of another value (Y). The predetermined range may beplus or minus 20%, 10%, 5%, 3%, 1%, 0.1%, or less than 0.1%, unlessotherwise indicated.

The indefinite articles “a” and “an,” as used in the specification andin the claims, unless clearly indicated to the contrary, should beunderstood to mean “at least one.” The phrase “and/or,” as used in thespecification and in the claims, should be understood to mean “either orboth” of the elements so conjoined, i.e., elements that areconjunctively present in some cases and disjunctively present in othercases. Multiple elements listed with “and/or” should be construed in thesame fashion, i.e., “one or more” of the elements so conjoined. Otherelements can optionally be present other than the elements specificallyidentified by the “and/or” clause, whether related or unrelated to thoseelements specifically identified. Thus, as a non-limiting example, areference to “A and/or B”, when used in conjunction with open-endedlanguage such as “comprising” can refer, in one embodiment, to A only(optionally including elements other than B); in another embodiment, toB only (optionally including elements other than A); in yet anotherembodiment, to both A and B (optionally including other elements); etc.

As used in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of or “exactly one of,” or, when used inthe claims, “consisting of,” will refer to the inclusion of exactly oneelement of a number or list of elements. In general, the term “or” asused shall only be interpreted as indicating exclusive alternatives(i.e. “one or the other but not both”) when preceded by terms ofexclusivity, such as “either,” “one of,” “only one of,” or “exactly oneof.” “Consisting essentially of,” when used in the claims, shall haveits ordinary meaning as used in the field of patent law.

As used in the specification and in the claims, the phrase “at leastone,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from one or more of theelements in the list of elements, but not necessarily including at leastone of each and every element specifically listed within the list ofelements and not excluding any combinations of elements in the list ofelements. This definition also allows that elements can optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

The use of “including,” “comprising,” “having,” “containing,”“involving,” and variations thereof, is meant to encompass the itemslisted thereafter and additional items.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed. Ordinal termsare used merely as labels to distinguish one claim element having acertain name from another element having a same name (but for use of theordinal term), to distinguish the claim elements.

EQUIVALENTS

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The foregoingembodiments are therefore to be considered in all respects illustrativerather than limiting on the invention described herein. Scope of theinvention is thus indicated by the appended claims rather than by theforegoing description, and all changes which come within the meaning andrange of equivalency of the claims are therefore intended to be embracedtherein.

1. A computer-implemented method comprising: obtaining one or moreuser-provided ratings for an item, wherein the one or more user-providedratings are provided by users of a computer system; determining whetherthe number of user-provided ratings for the item exceeds a thresholdnumber of ratings for the item; and based on a determination that thenumber of user-provided ratings for the item does not exceed thethreshold number of ratings: obtaining a plurality of predicted ratingsfor the item, and providing a set of ratings for the item to a device ofa user, wherein the set of ratings for the item includes the one or moreuser-provided ratings and the plurality of predicted ratings.
 2. Themethod of claim 1, wherein a distribution of the predicted ratings forthe item substantially matches a predicted distribution of ratings forthe item.
 3. The method of claim 2, further comprising determining thepredicted distribution of ratings for the item based, at least in part,on a categorization of the item, a manufacturer of the item, a seller ofthe item, and/or a price of the item.
 4. The method of claim 2, furthercomprising determining the predicted distribution of ratings for theitem based, at least in part, on actual distributions of ratings for oneor more other items.
 5. The method of claim 4, wherein the item and theone or more other items are in the same category or collection of items,are manufactured by the same manufacturer, are sold by the same seller,and/or have prices in the same range of prices.
 6. The method of claim1, wherein the one or more user-provided ratings are one or more firstuser-provided ratings, and wherein the method further comprises:obtaining one or more second user-provider ratings for the item;determining whether a combined number of the first and seconduser-provided ratings for the item exceeds the threshold number ofratings for the item; and based on a determination that the combinednumber of user-provided ratings for the item exceeds the thresholdnumber of ratings: providing a set of ratings for the item to a deviceof a user, wherein the set of ratings for the item consists of the oneor more first user-provided ratings and the one or more seconduser-provided ratings.
 7. A system comprising: one or more computersprogrammed to perform operations comprising: obtaining one or moreuser-provided ratings for an item, wherein the one or more user-providedratings are provided by users of a computer system; determining whetherthe number of user-provided ratings for the item exceeds a thresholdnumber of ratings for the item; and based on a determination that thenumber of user-provided ratings for the item does not exceed thethreshold number of ratings: obtaining a plurality of predicted ratingsfor the item, and providing a set of ratings for the item to a device ofa user, wherein the set of ratings for the item includes the one or moreuser-provided ratings and the plurality of predicted ratings.
 8. Thesystem of claim 7, wherein a distribution of the predicted ratings forthe item substantially matches a predicted distribution of ratings forthe item.
 9. The system of claim 8, wherein the operations furthercomprise determining the predicted distribution of ratings for the itembased, at least in part, on a categorization of the item, a manufacturerof the item, a seller of the item, and/or a price of the item.
 10. Thesystem of claim 8, wherein the operations further comprise determiningthe predicted distribution of ratings for the item based, at least inpart, on actual distributions of ratings for one or more other items.11. The system of claim 10, wherein the item and the one or more otheritems are in the same category or collection of items, are manufacturedby the same manufacturer, are sold by the same seller, and/or haveprices in the same range of prices.
 12. The system of claim 7, whereinthe one or more user-provided ratings are one or more firstuser-provided ratings, and wherein the operations further comprise:obtaining one or more second user-provider ratings for the item;determining whether a combined number of the first and seconduser-provided ratings for the item exceeds the threshold number ofratings for the item; and based on a determination that the combinednumber of user-provided ratings for the item exceeds the thresholdnumber of ratings: providing a set of ratings for the item to a deviceof a user, wherein the set of ratings for the item consists of the oneor more first user-provided ratings and the one or more seconduser-provided ratings.
 13. A computer storage medium having instructionsstored thereon that, when executed by a data processing apparatus, causethe data processing apparatus to perform operations comprising:obtaining one or more user-provided ratings for an item, wherein the oneor more user-provided ratings are provided by users of a computersystem; determining whether the number of user-provided ratings for theitem exceeds a threshold number of ratings for the item; and based on adetermination that the number of user-provided ratings for the item doesnot exceed the threshold number of ratings: obtaining a plurality ofpredicted ratings for the item, and providing a set of ratings for theitem to a device of a user, wherein the set of ratings for the itemincludes the one or more user-provided ratings and the plurality ofpredicted ratings.
 14. The computer storage medium of claim 13, whereina distribution of the predicted ratings for the item substantiallymatches a predicted distribution of ratings for the item.
 15. Thecomputer storage medium of claim 14, wherein the operations furthercomprise determining the predicted distribution of ratings for the itembased, at least in part, on a categorization of the item, a manufacturerof the item, a seller of the item, and/or a price of the item.
 16. Thecomputer storage medium of claim 14, wherein the operations furthercomprise determining the predicted distribution of ratings for the itembased, at least in part, on actual distributions of ratings for one ormore other items.
 17. The computer storage medium of claim 16, whereinthe item and the one or more other items are in the same category orcollection of items, are manufactured by the same manufacturer, are soldby the same seller, and/or have prices in the same range of prices. 18.The computer storage medium of claim 13, wherein the one or moreuser-provided ratings are one or more first user-provided ratings, andwherein the operations further comprise: obtaining one or more seconduser-provider ratings for the item; determining whether a combinednumber of the first and second user-provided ratings for the itemexceeds the threshold number of ratings for the item; and based on adetermination that the combined number of user-provided ratings for theitem exceeds the threshold number of ratings: providing a set of ratingsfor the item to a device of a user, wherein the set of ratings for theitem consists of the one or more first user-provided ratings and the oneor more second user-provided ratings.